Improving acceptability assessment for the labelling of affective speech corpora

In this paper we study how to address the assessment of affective speech corpora. We propose the use of several coefficients and provide guidelines to obtain a more complete background about the quality of their annotation. This proposal has been evaluated employing a corpus of non-acted emotions gathered from spontaneous interactions of users with a spoken dialogue system. The results show that, due to the nature of non-acted emotional corpora, traditional interpretations would in most cases consider the annotation of these corpora unacceptable even with very high inter-annotator agreement. Our proposal provides a basis to argue their acceptability by supplying a more fine-grained vision of their quality.